§1.1 Field of the Invention
The present invention concerns advertising, such as online advertising. In particular, the present invention concerns the review, such as the manual review, of advertisements for compliance with policies, such as quality assurance policies for example.
§1.2 Background Information
Advertising systems have employed policy guidelines to which ads must adhere in order to be served. Ensuring adherence with policies often entailed manual review. Unfortunately, manually reviewing all the ads, particularly in a large online ad network, is a significant challenge. For example, the continuing increase in ad submissions has forced online ad networks to allocate large amounts of human resources to manually review ads (often to the detriment of other areas such as optimization and customer support). Even when significant manual resources are allocated to reviewing ads, the backlog of pending ad reviews may nonetheless increase as online advertising continues to become more and more popular. Such backlogs can be a source of frustration to advertisers, and can represent lost opportunity for revenue to advertising networks.
In an effort to address manual ad review backlogs, auto-review systems have been implemented which may automate a portion of ad review. (See, e.g., U.S. patent application Ser. No. 11/026,415 (hereafter referred to as the '415 application and incorporated herein by reference), titled “POLICY CHECKING FRAMEWORK FOR AUTOMATED JUDGMENT OF ADVERTISEMENTS,” filed on Dec. 30, 2004, and listing Gregory Joseph Badros, Lucy Zhang and Robert J. Stets as inventors.) Despite the utility of auto-check systems, some or all of the automatically checked ads may still require manual review. Since ad submission rates are expected to continue to grow, ad review backlogs are expected to remain high. Consequently, backlogs of ads awaiting manual review are expected to be a continuing challenge.
As can be appreciated from the foregoing, it will be useful to improve the way in which ads, such as online ads for example, are reviewed.